Multi-neural net imaging apparatus and method

ABSTRACT

A multi-neural net imaging apparatus and method for classification of image elements, such as biological particles. The multi-net structure utilizes subgroups of particle features to partition the decision space by an attribute or physical characteristic of the particle and/or by individual and group particle classification that includes an unknown category. Preprocessing and post processing enables heuristic information to be included as part of the decision making process. Preprocessing classifies particles as artifacts based on certain physical characteristics. Post processing enables the use of contextual information either available from other sources or gleaned from the actual decision making process to further process the probability factors and enhance the decisions.

[0001] This application claims the benefit of U.S. ProvisionalApplication No. 60/199,237, filed Apr. 24, 2000, and entitledMULTI-NEURAL NET IMAGING APPARATUS AND METHOD.

TECHNICAL FIELD

[0002] The present invention relates to an imaging apparatus having aplurality of neural nets, and a method of training the neural nets and amethod of operating such an imaging apparatus. The apparatus can be usedto detect and classify biological particles and more particularly, fordetecting and classifying biological particles from human urine.

BACKGROUND OF THE INVENTION

[0003] Biological particle analysis apparatuses are well known in theart. See for example U.S. Pat. No. 4,338,024, assigned to the presentassignee, which describes a prior art machine that uses a computerhaving a stored fixed program to detect and to classify detectedbiological particles.

[0004] Standard decision theory that is used to sort biological particleimages is well known, and tends to sort particles by classification in aserial fashion. More specifically, for a urine sample containing aplurality of particle types, the particle images are searched for one ormore particle features unique to a single particle type, and thoseimages are extracted. This process is repeated for other particles oneparticle type at a time. The problem with this methodology is that eachparticle type can exhibit a range of values for the searched forparticle feature(s), and the range of values can overlap with those ofother particle types. There is also the problem of artifacts, which areparticle images that have no clinical significance, e.g. talc or hair,or cannot be classified due to the sensitivity of the imaging device orother problems with the image (e.g. boundary of particle undefined dueto partial capture). Artifact particle images need to be disregardedfrom the analysis in such a way as to not adversely affect the overallaccuracy of the particle analysis. Thus, it can be difficult toaccurately but reliably classify particles in a sample containingartifacts .

[0005] Most biological particle classification devices furthernecessitate manual manipulation to accurately classify the particles inthe sample. While particle features can be used to segregate particleimages by particle type, a trained user is needed to verify the result.

[0006] Neural net computers are also well known. The advantage of aneural net computer is its ability to “learn” from its experiences, andthus a neural net computer, in theory, can become more intelligent as itis trained.

[0007] There is a need for a biological particle classification methodand apparatus for accurate and automated classification of biologicalparticles in a sample, such as a urine sample.

SUMMARY OF THE INVENTION

[0008] In the present invention, a multi-neural net image detecting andclassification apparatus is disclosed. The multi-neural net moreefficiently uses the available information, which of necessity isfinite, in that it more effectively partitions the decision space,thereby allowing this information to be used to make fewer decisions ateach stage while still covering all outcomes with its totality ofdecisions. In addition, the neural net measures certainty at multiplestages of processing in order to force images to an abstention class,e.g. artifact. In some sense one can view this multi neural network asforcing the image data to run a gauntlet where at each stage of thegauntlet it is quite likely to be placed in an “I don't know” category.This is much more powerful than simply running through a single netbecause in essence what is accomplished is multiple fits of the data totemplates which are much better defined than a single template could bebecause of the more effective use of the information.

[0009] The present invention also relates to a large set of particlefeatures and a training method, which involves not simply a single passthrough the training set, but selecting from a number of nets and thenreducing the feature vector size. Finally, the present inventionprovides for preprocessing and post processing that enables heuristicinformation to be included as part of the decision making process. Postprocessing enables contextual information either available from othersources or gleaned from the actual decision making process to be used tofurther enhance the decisions.

[0010] The present invention is a method of classifying an element in animage into one of a plurality of classification classes, wherein theelement has a plurality of features. The method includes the steps ofextracting the plurality of features from the image of the element,determining a classification class of the element, and modifying thedetermined classification class of the element based upon a plurality ofpreviously determined classification class determinations. Thedetermination of the element classification class includes at least oneof:

[0011] selecting and processing a first subgroup of the extractedfeatures to determine a physical characteristic of the element, andselecting and processing a second subgroup of the extracted features inresponse to the physical characteristic determined to determine aclassification class of the element; and

[0012] selecting and processing a third subgroup of the extractedfeatures to determine a group of classification classes of the element,and selecting and processing a fourth subgroup of the extracted featuresin response to the determined classification class group to determine aclassification class of the element.

[0013] In other aspect of the present invention, an imaging apparatusfor classifying an element in an image into one of a plurality ofclassification classes, wherein the element has a plurality of features,includes means for extracting the plurality of features from the imageof the element, means for determining a classification class of theelement, and means for modifying the determined classification class ofthe element based upon a plurality of previously determinedclassification class determinations. The determining means includes atleast one of:

[0014] means for selecting and processing a first subgroup of theextracted features to determine a physical characteristic of theelement, and means for selecting and processing a second subgroup of theextracted features in response to the physical characteristic determinedto determine a classification class of the element, and

[0015] means for selecting and processing a third subgroup of theextracted features to determine a group of classification classes of theelement, and means for selecting and processing a fourth subgroup of theextracted features in response to the determined classification classgroup to determine a classification class of the element.

[0016] In yet another aspect of the present invention, a method ofclassifying an element in an image into one of a plurality ofclassifications, wherein the element has a plurality of features,includes the steps of extracting the plurality of features from theimage, determining a classification of the element based upon theplurality of features extracted by a first determination criteriawherein the first determination criteria includes the classification ofthe element as an unknown classification, determining a classificationof the element by a second determination criteria, different from thefirst determination criteria, in the event the element is classified asan unknown classification by the first determination criteria, anddetermining the classification of the element by a third determinationcriteria, different from the first and second determination criteria, inthe event the element is classified as one of a plurality ofclassifications by the first determination criteria.

[0017] In yet one more aspect of the present invention, an imagingapparatus for classifying an element in an image into one of a pluralityof classification classes, wherein the element has a plurality offeatures, includes: an extractor for extracting the plurality offeatures from the image of the element; a first processor thatdetermines a classification class of the element, and a second processorthat modifies the determined classification class of the element basedupon a plurality of previously determined classification classdeterminations. The first processor determines the classification classof the element by at least one of:

[0018] selecting and processing a first subgroup of the extractedfeatures to determine a physical characteristic of the element, andselecting and processing a second subgroup of the extracted features inresponse to the physical characteristic determined to determine aclassification class of the element; and

[0019] selecting and processing a third subgroup of the extractedfeatures to determine a group of classification classes of the element,and selecting and processing a fourth subgroup of the extracted featuresin response to the determined classification class group to determine aclassification class of the element.

[0020] Other objects and features of the present invention will becomeapparent by a review of the specification, claims and appended figures.

BRIEF DESCRIPTION OF DRAWINGS

[0021]FIG. 1 is a flow diagram showing the method of the presentinvention.

[0022]FIG. 2 is a schematic diagram of the apparatus of the presentinvention.

[0023]FIGS. 3A and 3B are flow diagrams illustrating the boundaryenhancement of the present invention.

[0024]FIG. 4 is a diagram illustrating the symmetry feature extractionof the present invention.

[0025]FIGS. 5A to 5D are drawings illustrating the skeletonization ofvarious shapes.

[0026]FIG. 6A is a flow diagram showing the LPF scan process of thepresent invention.

[0027]FIG. 6B is a flow diagram of the neural net classification usedwith the LPF scan process of the present invention.

[0028]FIG. 7A is a flow diagram showing the HPF scan process of thepresent invention.

[0029]FIG. 7B is a flow diagram of the neural net classification usedwith the HPF scan process of the present invention.

[0030]FIG. 8 is a schematic diagram of the neural net used with thepresent invention.

[0031]FIGS. 9A to 9C are tables showing the particle features used withthe various neural nets in the LPF and HPF scan processes of the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The present invention comprises a method and apparatus for makingdecisions about the classification of individual particle images in anensemble of images of biological particles for the purpose ofidentifying each individual image, and determining the number of imagesin each given class of particles.

[0033] Basic Method and Apparatus

[0034] The method is generally shown schematically in FIG. 1, andcomprises 5 basic steps:

[0035] 1) Collect individual images,

[0036] 2) Extract particle features from each individual image,

[0037] 3) Apply certain pre-processing rules to determineclassifications of individual images or how the classification processwill be performed,

[0038] 4) Classify the individual images using a multiple neural netdecision making structure, and

[0039] 5) Analyze the ensemble of decisions or a subset of the ensembleof decisions to determine the overall classification of the ensemble orchanges to classifications of certain subsets or individual images.

[0040] The method of the present invention further includes steps thattrain the individual neural nets used to make decisions, as well assteps that select the nets used in the final decision-making from amongmultiple nets produced by the training procedure.

[0041] There are three major hardware elements that are used toimplement the present invention: an imaging system 2, a first processor4 and a second processor 6. These hardware elements are schematicallyillustrated in FIG. 2.

[0042] Imaging system 2 is used to produce images of fields of view of asample containing the particles of interest. Imaging system 2 ispreferably a well known flow microscope as described in U.S. Pat. Nos.4,338,024, 4,393,466, 4,538,299 and 4,612,614, which are all herebyincorporated herein by reference. The flow microscope produces images ofsuccessive fields containing particles as they flow through a flow cell.

[0043] First processor 4 analyzes the images of successive fields, andisolates the particles in individual patches. A patch extractionapparatus (such as that described in U.S. Pat. Nos. 4,538,299 and5,625,709, which are hereby incorporated herein by reference) is used toanalyze the images produced by the imaging system and to define localareas (patches) containing particles of interest. The boundary of eachparticle is identified and defined, and used to extract the picture datafor each particle from the larger field, thereby producing digital patchimages that each contain the image of an individual particle of interest(resulting in a significant compression of the data subsequentlyrequired for processing). Imaging system 2 and first processor 4 combineto perform the first step (collection of individual images) shown inFIG. 1.

[0044] Second processor 6 analyzes each particle image to determine theclassification of the particle image. Second processor 6 performs thelast four steps shown in FIG. 1, as described below.

[0045] Boundary Enhancement—Mask Images

[0046] To enhance the particle feature extraction, the particle boundaryis further refined, and black and white mask images of the particles arecreated. This process effectively changes all the digital image pixelsoutside the boundary of the particle (background pixels) to blackpixels, and the pixels inside the particle boundary to white pixels. Theresulting white images of the particles against a black backgroundconveys the particles' shape and size very clearly, and are easy tooperate on for particle features based on shape and size only (giventhat the pixels are either white or black).

[0047] FIGS. 3A-3B illustrate the basic steps for transforming theparticle image into a mask image. First, a Shen-Castan edge detector (asdescribed in Parker, James R., Algorithms for Image Processing andComputer Vision, ISBN 0-471-14056-2, John Wiley & Sons, 1997, pp 29-32,and incorporated herein by reference) is used to define the edges ofparticles of interest, as illustrated in FIG. 3A. A particle image 10typically contains images of particles of interest 12 and otherparticles 14. The particle image 10 is smoothed, and a band limitedLaplacian image is created, followed by a gradient image. A thresholdroutine is used to detect the edges, whereby the locations where theintensity crosses a predetermined threshold are defined as edges. Thedetected edges are connected together to result in an edges image 16,which contains lines that correspond to detected boundaries that outlinethe various particles.

[0048] A mask image is created from the edge image 16 in the mannerillustrated in FIG. 3B. The edge image 16 is inverted so the boundarylines are white and the background is black. Then, the image is cleanedof all small specks and particles too small to be of interest. Smallgaps in the boundary lines are filled to connect some of the boundarylines together. The boundary lines are dilated to increase their width.This dilation is on the outer edges of the boundary lines, since theinner edges define the actual size of the particles. Disconnected pixelsare bridged to create complete lines that enclose the particles. Theinside of the boundaries are filled in to create blobs that representthe particles. The blobs are eroded to remove the pixels that had formedthe boundary lines, so that the blobs have the correct size. Finally,the largest blob is detected, and all the remaining blobs are discarded.The resulting image is a mask image of the particle, where the whiteblob against the black background accurately corresponds to the size andshape of the particle of interest.

[0049] Particle Feature Extraction

[0050] Once the image of a particle of interest has been localizedwithin a patch image, and its boundary further refined by creating awhite mask image of the particle, the patch and mask images are furtherprocessed in order to extract particle features (feature data) from theparticle image. Generally, the particle features numerically describethe size, shape, texture and color of the particles in numerousdifferent ways that aid in the accurate classification of particle type.The particle features can be grouped in families that are related to oneof these numerical descriptions, and can be extracted from the patchimage, the mask image, or both.

[0051] The first family of particle features all relate to the shape ofthe particle, which aid in differentiating red and white blood cellswhich tend to be round, crystals which tend to be square or rectangular,and casts with tend to be elongated. The first family of particlefeatures are:

[0052] 1. Particle Area: the number of pixels contained within theparticle boundary. Preferably, this particle feature is derived from themask image of the particle.

[0053] 2. Perimeter Length: the length of the particle boundary inpixels. Preferably, this is derived from the particle mask image bycreating a 4-neighborhood perimeter image of the mask, and counting thenumber of non-zero pixels.

[0054] 3. Shape Factor: an indication of the roundness of the particle.This is calculated as the square of the Perimeter Length, divided by theParticle Area.

[0055] 4. Area to Perimeter Ratio: another indication of the roundnessof the particle. This is calculated as the Particle Area divided by thePerimeter Length.

[0056] The second family of particle features relates to the symmetry ofthe particle, and in particular the determination of the number of linesof symmetry for any given shaped particle. This family of particlefeatures are quite useful in distinguishing casts (typically having aline of symmetry along its long axis) and squamous epithelial cells(SQEPs, which generally have no line of symmetry). This family ofparticle features utilizes information derived from line segmentsapplied at different angular orientations to the particle. Asillustrated in FIG. 4, a line segment 20 is drawn through the centroid22 of the mask image 19. For each point along the line segment 20, linesegments 24 a and 24 b perpendicular thereto are drawn to extend awayfrom the line segment 20 until they intersect the particle boundary, andthe difference in length of the opposing perpendicular line segments 24a and 24 b are calculated and stored. This calculation is repeated foreach point along the line segment 20, where all the difference valuesare then summed and stored as a Symmetry Value for line segment 20. Fora perfect circle, the Symmetry Value is zero for any line segment 20.The calculation of the Symmetry Value is then repeated for each angularrotation of line segment 20, resulting in a plurality of SymmetryValues, each one corresponding to a particular angular orientation ofline segment 20. The Symmetry Values are then normalized by the ParticleArea value, and sorted into an ordered list of Symmetry Values from lowto high.

[0057] The second family of particle features are:

[0058] 5. Minimum Symmetry: the lowest Symmetry Value in the orderedlist, which represents the maximum symmetry exhibited by the particle atsome value of rotation.

[0059] 6. 20% Symmetry: the Symmetry Value that constitutes the 20^(th)percentile of the ordered list of Symmetry Values.

[0060] 7. 50% Symmetry: the Symmetry Value that constitutes the 50^(th)percentile of the ordered list of Symmetry Values.

[0061] 8. 80% Symmetry: the Symmetry Value that constitutes the 80^(th)percentile of the ordered list of Symmetry Values.

[0062] 9. Maximum Symmetry: the highest Symmetry Value in the orderedlist, which represents the minimum symmetry exhibited by the particle atsome value of rotation.

[0063] 10. Average Symmetry: the average value of the Symmetry Values.

[0064] 11. Standard Deviation Symmetry: the standard deviation of theSymmetry Values.

[0065] The third family of particle features relate to skeletonizationof the particle image, which produces one or more line segments thatcharacterize both the size and the shape of the particle. These particlefeatures are ideal in identifying analytes having multiple components ina cluster, such as budding yeast, hyphae yeast, and white blood cellclumps. These analytes will have skeletons with multiple branches, whichare easy to differentiate from analytes having single branch skeletons.Creation of skeleton images is well known in the art of imageprocessing, and is disclosed in Parker, James R., Algorithms for ImageProcessing and Computer Vision, ISBN 0-471-14056-2, John Wiley & Sons,1997, pp 176-210), which is hereby incorporated herein by reference.Skeletonization essentially involves collapsing each portion of theparticle boundary inwardly in a direction perpendicular to itself. Forexample, a perfect circle collapses to a single point; a crescent mooncollapses to a curved line, a figure-8 collapses to 2 straight linesegments, and a cell with an indentation collapses to a curved line, asillustrated in FIGS. 5A-5D respectively. The preferred embodimentutilizes two skeletonization algorithms: ZSH and BZS. ZSH is theZhang-Suen thinning algorithm using Holt's variation plus staircaseremoval. BZS is the Zhang-Suen thinning algorithm using Holt'svariation. FIG. 5.11 in Parker (p. 182) shows the difference betweenresults when these algorithms are applied, along with C-code for eachalgorithm.

[0066] The third family of particle features are:

[0067] 12. ZSH Skeleton Size: the size of the skeleton, preferablydetermined by counting the number of pixels forming the skeleton. TheSkeleton Size for a perfect circle is 1, and for a crescent moon wouldbe the length of the curved line.

[0068] 13. ZSH Normalized Skeleton Size: Skeleton Size normalized by thesize of the particle, determined by dividing Skeleton Size by ParticleArea.

[0069] 14. BZS Skeleton Size: the size of the skeleton, preferablydetermined by counting the number of pixels forming the skeleton. TheSkeleton Size for a perfect circle is 1, and for a crescent moon wouldbe the length of the curved line.

[0070] 15. BZS Normalized Skeleton Size: Skeleton Size normalized by thesize of the particle, determined by dividing Skeleton Size by ParticleArea.

[0071] The fourth family of particle features relate to measuring theshape of the particle using radial lengths of radii that fit in theparticle, and the quantile rankings of these values. Specifically, acentroid is defined inside the particle, preferably using the maskimage, and a plurality of radii emanating from the centroid at differentangles extend out to the particle boundary. The lengths of the radii arecollected into a list of Radii Values, and the list is sorted from lowto high values. A certain % quantile of an ordered list of valuesrepresents the value having a position in the list that corresponds tothe certain percentage from the bottom of the list. For example, a 30%quantile of a list is the value that is positioned 30% up from bottom ofthe list, with 70% of the values being above it in the list. So, in anorder list of 10 values, the 30% quantile value is the seventh valuefrom the top of the list, and the 50% quantile is the median value ofthe list.

[0072] The fourth family of particle features are:

[0073] 16. 25% Radii Value: the value corresponding to the 25% quantileof the list of Radii Values.

[0074] 17. 50% Radii Value: the value corresponding to the 50% quantileof the list of Radii Values.

[0075] 18. 75% Radii Value: the value corresponding to the 75% quantileof the list of Radii Values.

[0076] 19. Smallest Mean Ratio: the ratio of the smallest Radii Value tothe mean Radii Value.

[0077] 20. Largest Mean Ratio: the ratio of the largest Radii Value tothe mean Radii Value.

[0078] 21. Average Radii Value: the average of the Radii Values.

[0079] 22. Standard Deviation Radii Value: the standard deviation of theRadii Values.

[0080] The fifth family of particle features measures the intensity ofthe particle image. Light absorption properties of different analytesdiffer significantly. For example, crystals are generally refractive andmay actually “concentrate” light so that their interior may be brighterthan the background. Stained white blood cells, however, will typicallybe substantially darker than the background. The average intensityreveals the overall light absorbing quality of the particle, while thestandard deviation of intensity measures the uniformity of theparticle's absorbing quality. In order to measure intensity, theparticle is preferably isolated by using the mask image in order to maskthe patch image of the particle. Thus, the only pixels left (inside themask) are those pixels contained inside the particle boundary. Thisfamily of particle features includes:

[0081] 23. Average Pixel Value: the average pixel value for all thepixels inside the particle boundary.

[0082] 24. Standard Deviation of Pixel Values: the standard deviation ofpixel values for pixels inside the particle boundary.

[0083] The sixth family of particle features use the Fourier Transformof the particle to measure the radial distribution of the particle. TheFourier Transform depends on the size, shape and texture (i.e. finegrain structure) of the particle. In addition to adding texture, theFourier Transform magnitude is independent of the position of theparticle, and particle rotation is directly reflected as a rotation ofthe transform. Finding clusters of energy at one rotation is anindication of linear aspects of the particle (i.e. the particle haslinear portions). This finding helps discriminate between particles suchas crystals versus red blood cells. The Fourier Transform of the patchimage of the particle is preferably calculated using a well known FastFourier Transform (FFT) algorithm with a window of 128×128 pixels. Thefollowing particle features are then calculated:

[0084] 25. FFT Average Intensity of Rotated 128 Pixel Line: a queuelisting of average pixel values along a 128 pixel line as a function ofrotation angle. This is calculated by placing a radial line of length128 pixels over the transform, and rotating the radial line through anarc of 180 degrees by increments of N degrees. For each increment of Ndegrees, the average of the pixel values along the radial line iscalculated. The average pixel values for the N degree increments arestored in a queue as Average Intensity along with the correspondingangular increment.

[0085] 26. FFT Maximum/Minimum 128 Pixel Angular Difference: thedifference between the angular values that correspond to the highest andlowest Average Intensity values stored in the queue.

[0086] 27. FFT 128 Pixel Average Intensity Standard Deviation: thestandard deviation of the Average Intensity values stored in the queue.

[0087] 28. FFT Average Intensity of Rotated 64 Pixel Line: same as theFFT Average Intensity of Rotated 128 Pixel Line, but instead using a 64pixel length radial line.

[0088] 29. FFT Maximum/Minimum 64 Pixel Angular Difference: same as theFFT Maximum/Minimum 128 Pixel Angular Difference, but instead using a 64pixel length radial line.

[0089] 30. FFT 64 Pixel Average Intensity Standard Deviation: same asthe FFT 128 Pixel Average Intensity Standard Deviation, but insteadusing a 64 pixel length radial line.

[0090] 31. FFT Average Intensity of Rotated 32 Pixel Line: same as theFFT Average Intensity of Rotated 128 Pixel Line, but instead using a 32pixel length radial line.

[0091] 32. FFT Maximum/Minimum 32 Pixel Angular Difference: same as theFFT Maximum/Minimum 128 Pixel Angular Difference, but instead using a 32pixel length radial line.

[0092] 33. FFT 32 Pixel Average Intensity Standard Deviation: same asthe FFT 128 Pixel Average Intensity Standard Deviation, but insteadusing a 32 pixel length radial line.

[0093] Additional FFT particle features all related to standarddeviation values based upon a rotated radial line of varying lengths areas follows:

[0094] 34. FFT 128 Pixel Average Intensity Standard Deviation Sort: asorted queue listing of the standard deviation of average pixel valuesalong a 128 pixel line for different rotations. This is calculated byplacing a radial line of length 128 pixels over the transform, androtating the radial line through an arc of 180 degrees by increments ofN degrees. For each increment of N degrees, the standard deviation valueof the pixels on the line is calculated. The standard deviation valuesfor all the N degree increments are sorted from low to high, and storedin a queue.

[0095] 35. FFT 128 Pixel Minimum Radial Standard Deviation: the minimumradial standard deviation value retrieved from the sorted queue listingof standard deviation values.

[0096] 36. FFT 128 Pixel Maximum Radial Standard Deviation: the maximumradial standard deviation value retrieved from the sorted queue listingof standard deviation values.

[0097] 37. FFT 128 Pixel 25% Quantile Radial Standard Deviation: theradial standard deviation value from the queue corresponding to the 25%quantile of the values stored in the queue.

[0098] 38. FFT 128 Pixel 50% Quantile Radial Standard Deviation: theradial standard deviation value from the queue corresponding to the 50%quantile of the values stored in the queue.

[0099] 39. FFT 128 Pixel 75% Quantile Radial Standard Deviation: theradial standard deviation value from the queue corresponding to the 75%quantile of the values stored in the queue.

[0100] 40. FFT 128 Pixel Minimum to Average Radial Standard DeviationRatio: the ratio of the minimum to average radial standard deviationvalues stored in the queue.

[0101] 41. FFT 128 Pixel Maximum to Average Radial Standard DeviationRatio: the ratio of the maximum to average radial standard deviationvalues stored in the queue.

[0102] 42. FFT 128 Pixel Average Radial Standard Deviation: the averageradial standard deviation value of the values stored in the queue.

[0103] 43. FFT 128 Pixel Standard Deviation of the Radial StandardDeviation: the standard deviation of all of the radial standarddeviation values stored in the queue.

[0104] 44. FFT 64 Pixel Average Intensity Standard Deviation Sort: thesame as the FFT 128 Pixel Average Intensity Standard Deviation Sort, butinstead using a 64 pixel length radial line.

[0105] 45. FFT 64 Pixel Minimum Radial Standard Deviation: the same asthe FFT 128 Pixel Minimum Radial Standard Deviation, but instead using a64 pixel length radial line.

[0106] 46. FFT 64 Pixel Maximum Radial Standard Deviation: the same asthe FFT 128 Pixel Maximum Radial Standard Deviation, but instead using a64 pixel length radial line.

[0107] 47. FFT 64 Pixel 25% Quantile Radial Standard Deviation: the sameas the FFT 128 Pixel 25% Quantile Radial Standard Deviation, but insteadusing a 64 pixel length radial line.

[0108] 48. FFT 64 Pixel 50% Quantile Radial Standard Deviation: the sameas the FFT 128 Pixel 50% Quantile Radial Standard Deviation, but insteadusing a 64 pixel length radial line.

[0109] 49. FFT 64 Pixel 75% Quantile Radial Standard Deviation: the sameas the FFT 128 Pixel 75% Quantile Radial Standard Deviation, but insteadusing a 64 pixel length radial line.

[0110] 50. FFT 64 Pixel Minimum to Average Radial Standard DeviationRatio: the same as the FFT 128 Pixel Minimum to Average Radial StandardDeviation Ratio, but instead using a 64 pixel length radial line.

[0111] 51. FFT 64 Pixel Maximum to Average Radial Standard DeviationRatio: the same as the FFT 128 Pixel Maximum to Average Radial StandardDeviation Ratio, but instead using a 64 pixel length radial line.

[0112] 52. FFT 64 Pixel Average Radial Standard Deviation: the same asthe FFT 128 Pixel Average Radial Standard Deviation, but instead using a64 pixel length radial line.

[0113] 53. FFT 64 Pixel Standard Deviation of the Radial StandardDeviation: the same as the FFT 128 Pixel Standard Deviation of theRadial Standard Deviation, but instead using a 64 pixel length radialline.

[0114] 54. FFT 32 Pixel Average Intensity Standard Deviation Sort: thesame as the FFT 128 Pixel Average Intensity Standard Deviation Sort, butinstead using a 32 pixel length radial line.

[0115] 55. FFT 32 Pixel Minimum Radial Standard Deviation: the same asthe FFT 128 Pixel Minimum Radial Standard Deviation, but instead using a32 pixel length radial line.

[0116] 56. FFT 32 Pixel Maximum Radial Standard Deviation: the same asthe FFT 128 Pixel Maximum Radial Standard Deviation, but instead using a32 pixel length radial line.

[0117] 57. FFT 32 Pixel 25% Quantile Radial Standard Deviation: the sameas the FFT 128 Pixel 25% Quantile Radial Standard Deviation, but insteadusing a 32 pixel length radial line.

[0118] 58. FFT 32 Pixel 50% Quantile Radial Standard Deviation: the sameas the FFT 128 Pixel 50% Quantile Radial Standard Deviation, but insteadusing a 32 pixel length radial line.

[0119] 59. FFT 32 Pixel 75% Quantile Radial Standard Deviation: the sameas the FFT 128 Pixel 75% Quantile Radial Standard Deviation, but insteadusing a 32 pixel length radial line.

[0120] 60. FFT 32 Pixel Minimum to Average Radial Standard DeviationRatio: the same as the FFT 128 Pixel Minimum to Average Radial StandardDeviation Ratio, but instead using a 32 pixel length radial line.

[0121] 61. FFT 32 Pixel Maximum to Average Radial Standard DeviationRatio: the same as the FFT 128 Pixel Maximum to Average Radial StandardDeviation Ratio, but instead using a 32 pixel length radial line.

[0122] 62. FFT 32 Pixel Average Radial Standard Deviation: the same asthe FFT 128 Pixel Average Radial Standard Deviation, but instead using a32 pixel length radial line.

[0123] 63. FFT 32 Pixel Standard Deviation of the Radial StandardDeviation: the same as the FFT 128 Pixel Standard Deviation of theRadial Standard Deviation, but instead using a 32 pixel length radialline.

[0124] Even more FFT particle features are used, all related to averagevalues based upon a rotated radial line of varying lengths:

[0125] 64. FFT 128 Pixel Average Intensity Sort: a sorted queue listingof the average pixel values along a 128 pixel line for differentrotations. This is calculated by placing a radial line of length 128pixels over the transform, and rotating the radial line through an arcof 180 degrees by increments of N degrees. For each increment of Ndegrees, the average value of the pixels on the line is calculated. Theaverage pixel values for all the N degree increments are sorted from lowto high, and stored in a queue.

[0126] 65. FFT 128 Pixel Minimum Average Value: the minimum radialaverage value retrieved from the sorted queue listing of average values.

[0127] 66. FFT 128 Pixel Maximum Radial Value: the maximum radialaverage value retrieved from the sorted queue listing of average values.

[0128] 67. FFT 128 Pixel 25% Quantile Radial Average Value: the radialaverage value from the queue corresponding to the 25% quantile of theaverage values stored in the queue.

[0129] 68. FFT 128 Pixel 50% Quantile Radial Average Value: the radialaverage value from the queue corresponding to the 50% quantile of theaverage values stored in the queue.

[0130] 69. FFT 128 Pixel 75% Quantile Radial Average Value: the radialaverage value from the queue corresponding to the 75% quantile of theaverage values stored in the queue.

[0131] 70. FFT 128 Pixel Minimum to Average Radial Average Value Ratio:the ratio of the minimum to average radial average values stored in thequeue.

[0132] 71. FFT 128 Pixel Maximum to Average Radial Average Value Ratio:the ratio of the maximum to average radial average values stored in thequeue.

[0133] 72. FFT 128 Pixel Average Radial Standard Deviation: the averageradial standard deviation value of the average values stored in thequeue.

[0134] 73. FFT 128 Pixel Standard Deviation of the Average Values: thestandard deviation of all of the radial average values stored in thequeue.

[0135] The seventh family of particle features use grayscale and colorhistogram quantiles of image intensities, which provide additionalinformation about the intensity variation within the particle boundary.Specifically, grayscale, red, green and blue histogram quantiles provideintensity characterization in different spectral bands. Further, stainsused with particle analysis cause some particles to absorb certaincolors, such as green, while others exhibit refractive qualities atcertain wavelengths. Thus, using all these particle features allows oneto discriminate between a stained particle such as white blood cellsthat absorb the green, and crystals that refract yellow light.

[0136] Histograms, cumulative histograms and quantile calculations aredisclosed in U.S. Pat. No. 5,343,538, which is hereby incorporatedherein by reference. The particle image is typically captured using aCCD camera that breaks down the image into three color components. Thepreferred embodiment uses an RGB camera that separately captures thered, green and blue components of the particle image. The followingparticle features are calculated based upon the grayscale, red, greenand blue components of the image:

[0137] 74. Grayscale Pixel Intensities: a sorted queue listing of thegrayscale pixel intensities inside the particle boundary. The grayscalevalue is a summation of the three color components. For each pixelinside the particle boundary (as masked by the mask image), thegrayscale pixel value is added to a grayscale queue, which is thensorted (e.g. from low to high).

[0138] 75. Minimum Grayscale Image Intensity: the minimum grayscalepixel value stored in the queue.

[0139] 76. 25% Grayscale Intensity: the value corresponding to the 25%quantile of the grayscale pixel values stored in the queue.

[0140] 77. 50% Grayscale Intensity: the value corresponding to the 50%quantile of the grayscale pixel values stored in the queue:

[0141] 78. 75% Grayscale Intensity: the value corresponding to the 75%quantile of the grayscale pixel values stored in the queue.

[0142] 79. Maximum Grayscale Image Intensity: the maximum grayscalepixel value stored in the queue.

[0143] 80. Red Pixel Intensities: a sorted queue listing of the redpixel intensities inside the particle boundary. The particle image isconverted so that only the red component of each pixel value remains.For each pixel inside the particle boundary (as masked by the maskimage), the red pixel value is added to a red queue, which is thensorted from low to high.

[0144] 81. Minimum Red Image Intensity: the minimum red pixel valuestored in the queue.

[0145] 82. 25% Red Intensity: the value corresponding to the 25%quantile of the red pixel values stored in the queue.

[0146] 83. 50% Red Intensity: the value corresponding to the 50%quantile of the red pixel values stored in the queue.

[0147] 84. 75% Red Intensity: the value corresponding to the 75%quantile of the red pixel values stored in the queue.

[0148] 85. Maximum Red Image Intensity: the maximum red pixel valuestored in the queue.

[0149] 86. Green Pixel Intensities: a sorted queue listing of the greenpixel intensities inside the particle boundary. The particle image isconverted so that only the green component of the pixel value remains.For each pixel inside the particle boundary (as masked by the maskimage), the green pixel value is added to a green queue, which is thensorted from low to high.

[0150] 87. Minimum Green Image Intensity: the minimum green pixel valuestored in the queue.

[0151] 88. 25% Green Intensity: the value corresponding to the 25%quantile of the green pixel values stored in the queue.

[0152] 89. 50% Green Intensity: the value corresponding to the 50%quantile of the green pixel values stored in the queue.

[0153] 90. 75% Green Intensity: the value corresponding to the 75%quantile of the green pixel values stored in the queue.

[0154] 91. Maximum Green Image Intensity: the maximum green pixel valuestored in the queue.

[0155] 92. Blue Pixel Intensities: a sorted queue listing of the bluepixel intensities inside the particle boundary. The particle image isconverted so that only the blue component of the pixel value remains.For each pixel inside the particle boundary (as masked by the maskimage), the blue pixel value is added to a blue queue, which is thensorted from low to high.

[0156] 93. Minimum Blue Image Intensity: the minimum blue pixel valuestored in the queue.

[0157] 94. 25% Blue Intensity: the value corresponding to the 25%quantile of the blue pixel values stored in the queue.

[0158] 95. 50% Blue Intensity: the value corresponding to the 50%quantile of the blue pixel values stored in the queue.

[0159] 96. 75% Blue Intensity: the value corresponding to the 75%quantile of the blue pixel values stored in the queue.

[0160] 97. Maximum Blue Image Intensity: the maximum blue pixel valuestored in the queue.

[0161] The eighth family of particle features use concentric circles andannuli to further characterize the variation in the FFT magnitudedistribution, which is affected by the size, shape and texture of theoriginal analyte image. A center circle is defined over a centroid ofthe FFT, along with seven annuli (in the shape of a washer) ofprogressively increasing diameters outside of and concentric with thecenter circle. The first annulus has an inner diameter equal to theouter diameter of the center circle, and an outer diameter that is equalto the inner diameter of the second annulus, and so on. The followingparticle features are calculated from the center circle and seven annuliover the FFT:

[0162] 98. Center Circle Mean Value: the mean value of the magnitude ofthe FFT inside the center circle.

[0163] 99. Center Circle Standard Deviation: the standard deviation ofthe magnitude of the FFT inside the center circle.

[0164] 100. Annulus to Center Circle Mean Value: the ratio of the meanvalue of the magnitude of the FFT inside the first annulus to that inthe center circle.

[0165] 101. Annulus to Center Circle Standard Deviation: the ratio ofthe standard deviation of the magnitude of the FFT inside the firstannulus to that in the center circle.

[0166] 102. Annulus to Circle Mean Value: the ratio of the mean value ofthe magnitude of the FFT inside the first annulus to that of a circledefined by the outer diameter of the annulus.

[0167] 103. Annulus to Circle Standard Deviation: the ratio of thestandard deviation of the magnitude of the FFT inside the first annulusto that of a circle defined by the outer diameter of the annulus.

[0168] 104. Annulus to Annulus Mean Value: the ratio of the mean valueof the magnitude of the FFT inside the first annulus to that of theannulus or center circle having the next smaller diameter (in the caseof the first annulus, it would be the center circle).

[0169] 105. Annulus to Annulus Standard Deviation: the ratio of thestandard deviation of the magnitude of the FFT inside the first annulusto that of the annulus or center circle having the next smaller diameter(in the case of the first annulus, it would be the center circle).

[0170] 106-111: Same as features 100-104, except the second annulus isused instead of the first annulus.

[0171] 112-117: Same as features 100-104, except the third annulus isused instead of the first annulus.

[0172] 118-123: Same as features 100-104, except the fourth annulus isused instead of the first annulus.

[0173] 124-129: Same as features 100-104, except the fifth annulus isused instead of the first annulus.

[0174] 130-135: Same as features 100-104, except the sixth annulus isused instead of the first annulus.

[0175] 136-141: Same as features 100-104, except the seventh annulus isused instead of the first annulus.

[0176] 154-197 is the same as 98-141, except they are applied to an FFTof the FFT of the particle image.

[0177] The last family of particle features use concentric squares withsides equal to 11%, 22%, 33%, 44%, 55%, and 66% of the FFT window size(e.g. 128) to further characterize the variation in the FFT magnitudedistribution, which is affected by the size, shape and texture of theoriginal analyte image. There are two well known texture analysisalgorithms that characterize the texture of an FFT. The first isentitled Vector Dispersion, which involves fitting a planar to teachregions using normals, and is described on pages 165-168 of Parker,which is incorporated by reference. The second is entitled SurfaceCurvature Metric, which involves conforming a polynomial to the region,and is described on pages 168-171 of Parker, which is incorporated byreference. The following particle features are calculated from differentsized windows over the FFT:

[0178] 142-147: Application of the Vector Dispersion algorithm to the11%, 22%, 33%, 44%, 55%, and 66% FFT windows, respectively.

[0179] 148-153: Application of the Surface Curvature Metric algorithm tothe 11%, 22%, 33%, 44%, 55%, and 66% FFT windows, respectively.

[0180] Processing and Decision Making

[0181] Once the foregoing particle features are computed, processingrules are applied to determine the classification of certain particlesor how all of the particles in the ensemble from the sample will betreated. The preferred embodiment acquires the particle images using alow power objective lens (e.g. 10×) to perform low power field (LPF)scans with a larger field of view to capture larger particles, and ahigh power objective lens (e.g. 40×) to perform high power field (HPF)scans with greater sensitivity to capture the more minute details ofsmaller particles.

[0182] The system of the present invention utilizes separate multineural net decision structures to classify particles captured in the LPFscan and HPF scan. Since most particles of interest will appear in oneof the LPF or HPF scans, but not both, the separate decision structuresminimize the number of particles of interest that each structure mustclassify.

[0183] Neural Net Structure

[0184]FIG. 8 illustrates the basic neural net structure used for all theneural nets in the LPF and HPF scans. The net includes an input layerwith inputs X₁ to X_(d), each corresponding to one of the particlefeatures described above that are selected for use with the net. Eachinput is connected to each one of a plurality of neurons Z₁ to Z_(J) ina hidden layer. Each of these hidden layer neurons Z₁ to Z_(J) sums allthe values received from the input layer in a weighted fashion, wherebythe actual weight for each neuron is individually assignable. Eachhidden layer neuron Z₁ to Z_(J) also applies a non-linear function tothe weighted sum. The output from each hidden layer neuron Z₁ to Z_(J)is supplied each one of a second (output) layer of neurons Y₁ to Y_(K).Each of the output layer neurons Y₁ to Y_(K) also sums the inputsreceived from the hidden layer in a weighted fashion, and applies anon-linear function to the weighted sum. The output layer neuronsprovide the output of the net, and therefore the number of these outputneurons corresponds to the number of decision classes that the netproduces. The number of inputs equals the number of particle featuresthat are chosen for input into the net.

[0185] As described later, each net is ‘trained’ to produce an accurateresult. For each decision to be made, only those particle features thatare appropriate to the decision of the net are selected for input intothe net. The training procedure involves modifying the various weightsfor the neurons until a satisfactory result is achieved from the net asa whole. In the preferred embodiment, the various nets were trainedusing NeuralWorks, product version 5.30, which is produced by NeuralWareof Carnegie, Pa., and in particular the Extended Delta Bar DeltaBack-propagation algorithm. The non-linear function used for all theneurons in all of the nets in the preferred embodiment is the hyperbolictangent function, where the input range is constrained between −0.8 and+0.8 to avoid the low slope region.

[0186] LPF Scan Process

[0187] The LPF scan process is illustrated in FIG. 6A, and starts bygetting the next particle image (analyte) using the low power objectivelens. A neural net classification is then performed, which involves theprocess of applying a cascading structure of neural nets to the analyteimage, as illustrated in FIG. 6B. Each neural net takes a selectedsubgroup of the calculated 198 particle features discussed above, andcalculates a classification probability factor ranging from zero to onethat the particle meets the criteria of the net. The cascadingconfiguration of the nets helps improve the accuracy of each neural netresult downstream, because each net can be specifically designed formore accuracy given that the particle types it operates on have beenprescreened to have or not have certain characteristics. For systemefficiency, all 198 particle features are preferably calculated for eachparticle image, and then the neural net classification process of FIG.6B is applied.

[0188] The first neural net applied to the particle image is the AMORClassifier Net, which decides whether or not the particle is amorphous.For the preferred embodiment, this net includes 42 inputs for a selectedsubset of the 198 particle features described above, 20 neurons in thehidden layer, and two neurons in the output layer. The column entitledLPF AMOR2 in the table of FIGS. 9A-9C shows the numbers of the 42particle features described above that were selected for use with thisnet. The first and second outputs of this net correspond to theprobabilities that the particle is or is not amorphous, respectively.Whichever probability is higher constitutes the decision of the net. Ifthe net decides the particle is amorphous, then the analysis of theparticle ends.

[0189] If it is decided that the particle is not amorphous, then theSQEP/CAST/OTHER Classifier Net is applied, which decides whether theparticle is a Squamous Epithelial cell (SQEP), a Cast cell (CAST), oranother type of cell. For the preferred embodiment, this net includes 48inputs for a selected subset of the 198 particle features describedabove, 20 neurons in the hidden layer, and three neurons in the outputlayer. The column entitled LPF CAST/SQEP/OTHER3 in the table of FIGS.9A-9C shows the numbers of the 48 particle features described above thatwere selected for use with this net. The first, second and third outputsof this net correspond to the probabilities that the particle a Cast, aSQEP, or another particle type, respectively. Whichever probability ishighest constitutes the decision of the net.

[0190] If it is decided that the particle is a Cast cell, then the CASTClassifier Net is applied, which decides whether the particle is a WhiteBlood Cell Clump (WBCC), a Hyaline Cast Cell (HYAL), or an unclassifiedcast (UNCC) such as a pathological cast cell . For the preferredembodiment, this net includes 36 inputs for a selected subset of the 198particle features described above, 10 neurons in the hidden layer, andthree neurons in the output layer. The column entitled LPF CAST3 in thetable of FIGS. 9A-9C shows the numbers of the 36 particle featuresdescribed above that were selected for use with this net. The first,second and third outputs of this net correspond to the probabilitiesthat the particle is a WBCC, HYAL or UNCC. Whichever probability ishighest constitutes the decision of the net.

[0191] If it is decided that the particle is a Squamous Epithelial cell,then the decision is left alone.

[0192] If it is decided that the particle is another type of cell, thenthe OTHER Classifier Net is applied, which decides whether the particleis a Non-Squamous Epithelial cell (NSE) such as a Renal Epithelial cellor a transitional Epithelial cell, an Unclassified Crystal (UNCX), Yeast(YEAST), or Mucus (MUCS). For the preferred embodiment, this netincludes 46 inputs for a selected subset of the 198 particle featuresdescribed above, 20 neurons in the hidden layer, and four neurons in theoutput layer. The column entitled LPF OTHER4 in the table of FIGS. 9A-9Cshows the numbers of the 46 particle features described above that wereselected for use with this net. The first, second, third and fourthoutputs of this net correspond to the probabilities that the particle isa NSE, UNCX, YEAST, or MUCS. Whichever probability is highestconstitutes the decision of the net.

[0193] Referring back to FIG. 6A, once the Neural Net Classification hasdecided the particle type, an ART by Abstention Rule is applied, todetermine if the particle should be classified as an artifact becausenone of the nets gave a high enough classification probability factor towarrant a particle classification. The ART by Abstention Rule applied bythe preferred embodiment is as follows: if the final classification bythe net structure is HYAL, and the CAST probability was less than 0.98at the SQEP/CAST/Other net, then the particle is reclassified as anartifact. Also, if the final classification by the net structure was aUNCC, and the CAST probability was less then 0.95 at the SQEP/CAST/Othernet, then the particle is reclassified as an artifact.

[0194] The next step shown in FIG. 6A applies to particles surviving theART by Abstention Rule. If the particle was classified by the netstructure as a UNCC, a HYAL or a SQEP, then that classification isaccepted unconditionally. If the particle was classified as another typeof particle, then a partial capture test is applied to determine if theparticle should be classified as an artifact. Partial capture testdetermines if the particle boundary hits one or more particle imagepatch boundaries, and thus only part of the particle image was capturedby the patch image. The partial capture test of the preferred embodimentbasically looks at the pixels forming the boundary of the patch toensure they represent background pixels. This is done by collecting acumulative intensity histogram on the patch boundaries, and calculatingLower and Upper limits of these intensities. The Lower limit in thepreferred embodiment is either the third value from the bottom of thehistogram, or the value 2% from the bottom of the histogram, whicheveris greater. The Upper limit is either the third value from the top ofthe histogram, or the value 2% from the top of the histogram, whicheveris greater. The patch image is deemed a partial capture if the lowerlimit is less than 185 (e.g. of a pixel intensity that ranges from 0 to255). The patch is also deemed a partial capture if the upper limit isless than or equal to 250 and the lower limit is less than 200 (this isto take care of the case where the halo of a particle image touches thepatch image boundary). All particles surviving the partial capture testmaintain their classification, and the LPF scan process is complete.

[0195] In the preferred embodiment, the partial capture test is alsoused as one of the particle features used by some of the neural nets.The feature value is 1 if the particle boundary is found to hit one ormore particle image patch boundaries, and a zero if not. This particlefeature is numbered “0” in FIGS. 9A-9C.

[0196] HPF Scan Process

[0197] The HPF scan process is illustrated in FIG. 7A, and starts bygetting the next particle image (analyte) using the high power objectivelens. Two pre-processing artifact classification steps are performedbefore submitting the particles to neural net classification. The firstpreprocessing step begins by defining five size boxes (HPF1-HPF5), witheach of the particles being associated with the smallest box that it cancompletely fit in to. In the preferred embodiment, the smallest box HPF5is 12 by 12 pixels, and the largest box HPF1 is 50 by 50 pixels. Allparticles associated with the HPF5 box are classified as an artifact andremoved from further consideration, because those particles are toosmall for accurate classification given the resolution of the system.

[0198] The second pre-processing step finds all remaining particles thatare associated with the HPF3 or HPF4 boxes, that have a cell area thatis less than 50 square pixels, and that are not long and thin, andclassifies them as artifacts. This second preprocessing step combinessize and aspect ratio criteria, which eliminates those smaller particleswhich tend to be round. Once particles associated with the HPF3 or HPF4boxes and with a cell area under 50 square pixels have been segregated,each such particle is classified as an artifact if either of thefollowing two criteria are met. First, if the square of the particleperimeter divided by the particle area is less than 20, then theparticle is not long and thin and is classified an artifact. Second, ifthe ratio of eigenvalues of the covariance matrix of X and Y moments(which is also called the Stretch Value) is less than 20, then theparticle is not long and thin and is classified an artifact.

[0199] Particle images that survive the two preprocessing stepsdescribed above are subjected to the cascading structure of neural netsillustrated in FIG. 7B. Each neural net takes a selected subgroup of thecalculated 198 particle features discussed above, and calculates aclassification probability factor ranging from zero to one that theparticle meets the criteria of net. As with the cascading configurationof the nets, this helps improve the accuracy of each neural net resultdownstream, and preferably all 198 particle features are calculated foreach particle image before the HPF scan commences.

[0200] The first neural net applied to the particle image is the AMORClassifier Net, which decides whether or not the particle is amorphous.For the preferred embodiment, this net includes 50 inputs for a selectedsubset of the 198 particle features described above, 10 neurons in thehidden layer, and two neurons in the output layer. The column entitledHPF AMOR2 in the table of FIGS. 9A-9C shows the numbers of the 50particle features described above that were selected for use with thisnet. The first and second outputs of this net correspond to theprobabilities that the particle is or is not amorphous. Whicheverprobability is higher constitutes the decision of the net. If the netdecides the particle is amorphous, then the analysis of the particleends.

[0201] If it is decided that the particle is not amorphous, then theRound/Not Round Classifier Net is applied, which decides whether theparticle shape exhibits a predetermined amount of roundness. For thepreferred embodiment, this net includes 39 inputs for a selected subsetof the 198 particle features described above, 20 neurons in the hiddenlayer, and two neurons in the output layer. The column entitled HPFROUND/NOT ROUND2 in the table of FIGS. 9A-9C shows the numbers of the 39particle features described above that were selected for use with thisnet. The first and second outputs of this net correspond to theprobabilities that the particle is or is not ‘round’. Whicheverprobability is highest constitutes the decision of the net.

[0202] If it is decided that the particle is ‘round’, then the RoundCells Classifier Net is applied, which decides whether the particle is aRed Blood Cell (RBC), a White Blood Cell (WBC), a Non-SquamousEpithelial cell (NSE) such as a Renal Epithelial cell or a transitionalEpithelial cell, or Yeast (YEAST). For the preferred embodiment, thisnet includes 18 inputs for a selected subset of the 198 particlefeatures described above, 3 neurons in the hidden layer, and threeneurons in the output layer. The column entitled HPF Round4 in the tableof FIGS. 9A-9C shows the numbers of the 18 particle features describedabove that were selected for use with this net. The first, second, thirdand fourth outputs of this net correspond to the probabilities that theparticle is a RBC, a WBC, a NSE or YEAST, respectively. Whicheverprobability is highest constitutes the decision of the net.

[0203] If it is decided that the particle is not ‘round’, then the NotRound Cells Classifier Net is applied, which decides whether theparticle is a Red Blood Cell (RBC), a White Blood Cell (WBC), aNon-Squamous Epithelial cell (NSE) such as a Renal Epithelial cell or atransitional Epithelial cell, an Unclassified Crystal (UNCX), Yeast(YEAST), Sperm (SPRM) or Bacteria (BACT). For the preferred embodiment,this net includes 100 inputs for a selected subset of the 198 particlefeatures described above, 20 neurons in the hidden layer, and sevenneurons in the output layer. The column entitled HPF NOT ROUND7 in thetable of FIGS. 9A-9C shows the numbers of the 100 particle featuresdescribed above that were selected for use with this net. The sevenoutputs of this net correspond to the probabilities that the particle isa RBC, a WBC, a NSE, a UNCX, a YEAST, a SPRM or a BACT. Whicheverprobability is highest constitutes the decision of the net.

[0204] Referring back to FIG. 7A, once the Neural Net Classification hasdecided the particle type, an ART by Abstention Rule is applied, todetermine if the particle should be classified as an artifact becausenone of the nets gave a high enough classification probability factor towarrant a particle classification. The ART by Abstention Rule applied bythe preferred embodiment reclassifies four types of particles asartifacts if certain criteria are met. First, if the finalclassification by the net structure is Yeast, and the YEAST probabilitywas less than 0.9 at the Not Round Cells Classification Net, then theparticle is reclassified as an artifact. Second, if the finalclassification by the net structure was a NSE, and the NSE probabilitywas less than 0.9 at the Round Cells Classifier Net, or the roundprobability was less than 0.9 at the Round/Not Round Classifier Net,then the particle is reclassified as an artifact. Third, if the finalclassification by the net structure was a not round NSE, and the NSEprobability was less than 0.9 at the Not Round Cells Classifier Net,then the particle is reclassified as an artifact. Fourth, if the finalclassification by the net structure was a UNCX, and the UNCX probabilitywas less than 0.9 at the Not Round Cells Classifier Net, or the roundprobability was less than 0.9 at the Round/Not Round Classifier Net,then the particle is reclassified as an artifact.

[0205] The next step shown in FIG. 7A is a partial capture test, whichis applied to all particles surviving the ART by Abstention Rule. Thepartial capture test determines if the particle should be classified asan artifact because the particle boundary hits one or more particleimage patch boundaries, and thus only part of the particle image wascaptured by the patch image. As with the LPF scan, the partial capturetest of the preferred embodiment basically looks at the pixels formingthe boundary of the patch to ensure they represent background pixels.This is done by collecting a cumulative intensity histogram on the patchboundaries, and calculating lower and upper limits of these intensities.The lower limit in the preferred embodiment is either the third valuefrom the bottom of the histogram, or the value 2% from the bottom of thehistogram, whichever is greater. The upper limit is either the thirdvalue from the top of the histogram, or the value 2% from the top of thehistogram, whichever is greater. The patch image is deemed a partialcapture if the lower limit is less than 185 (e.g. of a pixel intensitythat ranges from 0 to 255). The patch is also deemed a partial captureif the upper limit is less than or equal to 250 and the lower limit isless than 200 to take care of the case where the halo of a particleimage touches the patch image boundary.

[0206] All particles surviving the partial capture test maintain theirclassification. All particles deemed a partial capture are furthersubjected to an ART by Partial Capture Rule, which reclassifies suchparticles as an artifact if any of the following 6 criteria are met:

[0207] 1. If it was associated with the HPF1 size box.

[0208] 2. If it was not classified as either a RBC, WBC, BYST, OR UNCX.

[0209] 3. If it was classified as a RBC, and if it was associated withthe HPF2 size box or had a Stretch Value greater than or equal to 5.0.

[0210] 4. If it was classified as a WBC, and had a Stretch Value greaterthan or equal to 5.0.

[0211] 5. If it was classified as a UNCX, and had a Stretch Valuegreater than or equal to 10.0.

[0212] 6. If it was classified as a BYST, and had a Stretch Valuegreater than or equal to 20.0.

[0213] If none of these six criteria are met by the particle image, thenthe neural net classification is allowed to stand even though theparticle was deemed a partial capture, and the HPF scan process iscomplete. These six rules were selected to keep particle classificationdecisions in cases where partial capture does not distort the neural netdecision making process, while eliminating those particles where apartial capture would likely lead to an incorrect decision.

[0214] To best determine which features should be used for each of theneural nets described above, the feature values input to any givenneural net are modified one at a time by a small amount, and the effecton the neural net output is recorded. Those features having the greatestaffect on the output of the neural net should be used.

[0215] Post Processing Decision Making

[0216] Once all the particles images are classified by particle type,post decision processing is performed to further increase the accuracyof the classification results. This processing considers the completeset of results, and removes classification results that as a whole arenot considered trustworthy.

[0217] User settable concentration thresholds are one type of postdecision processing that establishes a noise level threshold for theoverall results. These thresholds are settable by the user. If theneural net classified image concentration is lower than the threshold,then all the particles in the category are reclassified as artifacts.For example, if the HPF scan finds only a few RBC's in the entiresample, it is likely these are erroneous results, and these particlesare reclassified as artifacts.

[0218] Excessive amorphous detection is another post decision processthat discards questionable results if too many particles are classifiedas amorphous. In the preferred embodiment, if there are more than 10non-amorphous HPF patches, and more than 60% of them are classified tobe amorphous by the neural nets, then the results for the entirespecimen are discarded as unreliable.

[0219] The preferred embodiment also includes a number of LPF falsepositive filters, which discard results that are contradictory orquestionable. Unlike HPF particles, LPF artifacts come in all shapes andsizes. In many cases, given the resolution of the system, it isimpossible to distinguish LPF artifacts from true clinically significantanalytes. In order to reduce false positives due to LPF artifacts, anumber of filters are used to look at the aggregate decisions made bythe nets, and discard those results that simply make no sense. Forexample, if the HPF WBC count is less than 9, then all LPF WBCC's shouldbe reclassified as artifacts, since clumps of white blood cells areprobably not present if white blood cells are not found in significantnumbers. Further, the detection of just a few particles of certain typesshould be disregarded, since it is unlikely that these particles arepresent in such low numbers. In the preferred embodiment, the systemmust find more than 3 LPF UNCX detected particles, or more than 2 LPFNSE detected particles, or more than 3 LPF MUC detected particles, ormore than 2 HPF SPRM detected particles, or more than 3 LPF YEASTdetected particles. If these thresholds are not met, then the respectiveparticles are re-classified as artifacts. Moreover, there must be atleast 2 HPF BYST detected particles to accept any LPF YEAST detectedparticles.

[0220] Neural Net Training and Selection

[0221] Each neural net is trained using a training set of pre-classifiedimages. In addition to the training set, a second smaller set ofpre-classified images is reserved as the test set. In the preferredembodiment, the commercial program NeuralWare, published by NeuralWorks,is used to perform the training. Training stops when the average erroron the test set is minimized.

[0222] This process is repeated for multiple starting seeds and netstructures (i.e. number of hidden layers and elements in each layer).The final choice is based not only on the overall average error rate,but also to satisfy constraints on errors between specific classes. Forexample, it is undesirable to identify a squamous epithelial cell as apathological cast because squamous epithelial cells occur normally infemale urine specimens, but pathological casts would indicate anabnormal situation. Therefore, the preferred embodiment favors nets withSQEP to UNCC error rates less than 0.03 at the expense of a greatermisclassification rate of UNCC as SQEP. This situation somewhatdecreases the sensitivity to UNCC detection, but minimizes falsepositives in female specimens, which with sufficiently high rate ofoccurrence would render the system useless since a high proportion offemale urine samples would be called abnormal. Thus, it is preferable toemploy a method that not only minimizes the overall error rate, but alsoconsiders the cost of specific error rates in the selection of the“optimal” nets, and build this selection criterion into the nettraining.

[0223] As can be seen from the forgoing, the method and apparatus of thepresent invention differs from the prior art in the following respect.In the prior art, at each stage of processing, a classification of aparticle is made, with the remaining unclassified particles consideredartifacts or unknowns. In order to minimize the classification ofparticles as artifacts or unknowns, the range of values forclassification at each stage is large. This can cause misclassificationof particles.

[0224] In contrast, the range of values for classification at each stageof the present invention is narrow, resulting in only particles havinggreater probability of certainty being so classified, and the remainderbeing classified in a classification for further processing that isrelated to the previous stage of processing. The multi-net structure ofthe present invention utilizes subgroups of the particle features topartition the decision space by an attribute or physical characteristicof the particle (e.g. its roundness) and/or by individual and groupparticle classification that includes an unknown category. Thispartitioned decision space, which produces probability factors at eachdecision, more efficiently uses the available information, which ofnecessity is finite, and effectively allows this information to be usedto make the same number of total decisions, but with fewer possibleoutcomes at each stage. Preprocessing and post processing enablesheuristic information to be included as part of the decision makingprocess. Post processing enables the use of contextual informationeither available from other sources or gleaned from the actual decisionmaking process to further process the probability factors and enhancethe decisions. The use of neural net certainty measures at multiplestages of processing forces images to an abstention class, i.e.artifact. In some sense one can view this multi network approach asforcing the image data to run a gauntlet where at each stage of thegauntlet it is quite likely to be placed in an “I don't know” category.This is much more powerful than simply running through a single netbecause in essence what is accomplished is multiple fits of the data totemplates which are much better defined than a single template could be,which allows effective use of the information. Another way to thinkabout this is that data in different subspaces is analyzed, and isrequired it to fit perfectly in some sense, or well enough, with thecharacteristics of that subspace or else it falls out of the race. Thetraining method of the present invention involves not simply a singlepass through the training set, but selecting from a number of nets andthen reducing the feature vector size. The high number of featuresthemselves, each focussing on a particular set of physicalcharacteristics, increases the accuracy the system.

[0225] It is to be understood that the present invention is not limitedto the embodiments described above and illustrated herein, butencompasses any and all variations falling within the scope of theappended claims. Therefore, it should be understood that while thepresent invention is described with respect to the classification ofimages of biological samples, it also includes image analysis for anyimage having features that can be extracted and used to classify theimage. For example, the present invention can be used for facialrecognition. Features can be extracted to identify and classify theshape, size, location and dimension of the eyes, nose, mouth, etc., ormore general features such as face shape and size, so that the facialimages can be identified and classified into predeterminedclassifications.

What is claimed is:
 1. A method of classifying an element in an imageinto one of a plurality of classification classes, wherein the elementhas a plurality of features, the method comprising the steps of:extracting the plurality of features from the image of the element;determining a classification class of the element by at least one of:selecting and processing a first subgroup of the extracted features todetermine a physical characteristic of the element, and selecting andprocessing a second subgroup of the extracted features in response tothe physical characteristic determined to determine a classificationclass of the element; and selecting and processing a third subgroup ofthe extracted features to determine a group of classification classes ofthe element, and selecting and processing a fourth subgroup of theextracted features in response to the determined classification classgroup to determine a classification class of the element; and modifyingthe determined classification class of the element based upon aplurality of previously determined classification class determinations.2. The method of claim 1, wherein the element is a biological particle.3. The method of claim 1, wherein each of the determinations includesassigning a probability factor, and further including the step ofmodifying the determined classification class to an artifactclassification in the event one or more of the probability factors usedto classify the element fails to exceed a predetermined threshold value.4. The method of claim 1, further comprising the step of: classifyingthe element as an artifact based on a physical characteristic of theelement, wherein the artifact element bypasses the determination of theclassification class of the element.
 5. The method of claim 1, furthercomprising the steps of: determining whether a boundary of the elementintersects a border of an image containing the element, and modifyingthe determined classification class of the element to an artifactclassification in the event the element boundary and image border aredetermined to intersect.
 6. The method of claim 1, wherein theprocessing of the first, second, third and fourth subgroups of theextracted features is performed using neural nets.
 7. The method ofclaim 6, further comprising the steps of: training the neural nets byselecting and processing the first, second, third and fourth subgroupsof the extracted features using a training set of known elements alongwith a test set of elements, wherein the training of the neural nets isrepeatedly performed until the accuracy rate of the determination ofclassification class of the test set of elements reaches a predeterminedvalue.
 8. The method of claim 6, wherein the first, second, third andfourth subgroups of the plurality of features are selected by modifyingeach of the feature values by a predetermined amount, and selectingthose features that affect the output of the respectively neural net themost.
 9. The method of claim 1, wherein one of the plurality ofextracted features is symmetry of the element, and the extraction of thesymmetry feature includes: defining a first line segment that crosses acentroid of the element; defining a second and third line segments forpoints along the first line segment that orthogonally extend away fromthe first line segment in opposite directions; utilizing the differencebetween the lengths of the second and third line segments to calculatethe extracted symmetry feature of the element.
 10. The method of claim1, wherein one of the plurality of extracted features is skeletonizationof the element image, and the extraction of the skeletonization featureincludes orthogonally collapsing a boundary of the element to form oneor more line segments.
 11. The method of claim 1, wherein at least oneof the plurality of extracted features is a measure of a spatialdistribution of the element image, and at least another one of theplurality of extracted features is a measure of a spatial frequencydomain of the element image.
 12. An imaging apparatus for classifying anelement in an image into one of a plurality of classification classes,wherein the element has a plurality of features, the apparatuscomprising: means for extracting the plurality of features from theimage of the element; means for determining a classification class ofthe element, the determining means including at least one of: means forselecting and processing a first subgroup of the extracted features todetermine a physical characteristic of the element, and means forselecting and processing a second subgroup of the extracted features inresponse to the physical characteristic determined to determine aclassification class of the element; and means for selecting andprocessing a third subgroup of the extracted features to determine agroup of classification classes of the element, and means for selectingand processing a fourth subgroup of the extracted features in responseto the determined classification class group to determine aclassification class of the element; and means for modifying thedetermined classification class of the element based upon a plurality ofpreviously determined classification class determinations.
 13. Theapparatus of claim 12, wherein the element is a biological particle. 14.The apparatus of claim 12, wherein each of the determinations includesassigning a probability factor, and wherein the determining meansfurther includes means for modifying the determined classification classto an artifact classification in the event one or more of theprobability factors used to classify the element fails to exceed apredetermined threshold value.
 15. The apparatus of claim 12, furthercomprising: means for classifying the element as an artifact based on aphysical characteristic of the element, wherein the artifact elementbypasses the determining means.
 16. The apparatus of claim 12, furthercomprising: means for determining whether a boundary of the elementintersects a border of an image containing the element, and means formodifying the determined classification class of the element to anartifact classification in the event the element boundary and imageborder are determined to intersect.
 17. The apparatus of claim 12,wherein the processing of the first, second, third and fourth subgroupsof the extracted features is performed using neural nets.
 18. Theapparatus of claim 17, further comprising: means for training the neuralnets by selecting and processing the first, second, third and fourthsubgroups of the extracted features using a training set of knownelements along with a test set of elements, wherein the training meansrepeatedly trains the neural nets until the accuracy rate of thedetermination of classification class of the test set of elementsreaches a predetermined value.
 19. The apparatus of claim 17, whereinthe first, second, third and fourth subgroups of the plurality offeatures are selected by modifying each of the feature values by apredetermined amount, and selecting those features that affect theoutput of the respectively neural net the most.
 20. The apparatus ofclaim 12, wherein one of the plurality of extracted features is symmetryof the element, and the extraction means includes: means for defining afirst line segment that crosses a centroid of the element; means fordefining a second and third line segments for points along the firstline segment that orthogonally extend away from the first line segmentin opposite directions; means for utilizing the difference between thelengths of the second and third line segments to calculate the extractedsymmetry feature of the element.
 21. The apparatus of claim 12, whereinone of the plurality of extracted features is skeletonization of theelement image, and the extraction means further includes means fororthogonally collapsing a boundary of the element to form one or moreline segments.
 22. The apparatus of claim 12, wherein at least one ofthe plurality of extracted features is a measure of a spatialdistribution of the element image, and at least another one of theplurality of extracted features is a measure of a spatial frequencydomain of the element image.
 23. A method of classifying an element inan image into one of a plurality of classifications, wherein the elementhas a plurality of features, the method comprising the steps of:extracting the plurality of features from the image; determining aclassification of the element based upon the plurality of featuresextracted by a first determination criteria, wherein the firstdetermination criteria includes the classification of the element as anunknown classification; determining a classification of the element by asecond determination criteria, different from the first determinationcriteria, in the event the element is classified as an unknownclassification by the first determination criteria; and determining theclassification of the element by a third determination criteria,different from the first and second determination criteria, in the eventthe element is classified as one of a plurality of classifications bythe first determination criteria.
 24. An imaging apparatus forclassifying an element in an image into one of a plurality ofclassification classes, wherein the element has a plurality of features,the apparatus comprising: an extractor for extracting the plurality offeatures from the image of the element; a first processor thatdetermines a classification class of the element by at least one of:selecting and processing a first subgroup of the extracted features todetermine a physical characteristic of the element, and selecting andprocessing a second subgroup of the extracted features in response tothe physical characteristic determined to determine a classificationclass of the element; and selecting and processing a third subgroup ofthe extracted features to determine a group of classification classes ofthe element, and selecting and processing a fourth subgroup of theextracted features in response to the determined classification classgroup to determine a classification class of the element; and a secondprocessor that modifies the determined classification class of theelement based upon a plurality of previously determined classificationclass determinations.